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determining a temporal difference image value based 
upon said correlation output values . 




REMARKS 



This amendment is in response to the Office Action dated 
4/9/03. Also enclosed are a Terminal Disclaimer and an Associate 
Power of Attorney. Entry of this Amendment and reconsideration of 
this application are respectfully requested. The amendments are 
also given in the older format, with insertions and deletions, in 
Attachment A following these remarks. 

Objections to the Specification 

Each of the Examiner's objections to the specification have 
been addressed. Each requested correction has been made, as 
indicated in the amended paragraphs above . 

Claim 16 was objected to due to a typographical error. 
Correction has been made. 

Objections to the Drawings 

The Examiner correctly noted that reference number "82" in 
figure 4 should be changed to u 22" . The Examiner also noted that 
FIG. 7 was not included with the specification. 

In response, a new set of drawings is enclosed: the 
correction to FIG. 4 has been made, and FIG. 7 is included in the 
set . 

Double Patenting 

Claims 1-22 were rejected under the judicially-created 
doctrine of obviousness -type double patenting, over a patent to 
Burke et al . (U.S. Patent No. 6,351,660), or over Burke in 
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combination with one or more other patents. 

In response, a terminal disclaimer in compliance with 37 
C.F.R. §1.321 (c) is enclosed, which disclaims the terminal part 
of the statutory term of any patent granted on the instant 
application, which would extend beyond the expiration date of the 
full statutory term of U.S. Patent No. 6,351,660. Note that U.S. 
Patent No. 6,351,660 and the present application are commonly 
assigned to Litton Systems, Inc. 

It is believed that the filing of the enclosed terminal 
disclaimer overcomes the Examiner's double patenting rejections. 

Claim Rejections 
Rejections under §102 

Claims 1 and 9-10 were rejected as anticipated by a patent 
to Yanagita et al . 

Claim 1 

In response, claim 1 has been amended to better clarify the 
differences between the applicants' invention and that which is 
disclosed in Yanagita. 

As amended, claim 1 is directed to a method of visually 
documenting historical changes in biological tissue. The method 
comprises the steps of: 

(a) obtaining a first image of a region of tissue; 

(b) after obtaining the first image, obtaining a second 
image of the region of tissue; 

(c) digitally storing the first and second images as 
digitized first and second images; 

(d) spatially adjusting at least one of the first and second 
digitized images to spatially register the images so that 
corresponding features in both images are mapped to corresponding 
positions; and 
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(e) creating from the first and second digitized images a 
derived image which visually emphasizes temporal differences 
between the first and second images, thereby visually emphasizing 
historical changes between the images. 

Support for this amendment is found in the specification at, 
for example, page 7, lines 21-24. 

The applicants' usage of "temporal differences" as recited 
in claim 1 is essential to their method, and represents a very 
significant distinction with respect to the cited art. "Temporal 
difference" learning is a particular technique known to those 
familiar with the art of tracking historical changes using time- 
separated images (see, e.g., http : //www- 
anw . cs . umass . edu/ ~rich/book/the-book . html (MIT Press, Cambridge, 
MA 1998) for a discussion of temporal difference learning) . A 
temporal difference image provides a scalar or reinforcement 
metric of learning; it is a reinforcement feedback signal 
indicating a scalar of how good/bad, how much change (vector) , or 
how strongly (acceleration) an action is happening. The scalars 
may represent, for example, a natural change, or the 
effectiveness of a treatment. For example, if digitized two 
dimensional images are considered, the images are preferably 
first normalized to a common average value, then corresponding 
pixels are subtracted and the temporal difference values stored 
in a corresponding position in a temporal difference image. 
Analogously, if three-dimensional imagery is available (which is 
greatly preferred) , voxels at corresponding locations in the 
early and later images are subtracted one from the other, and the 
result stored in a voxel of a temporal difference (volume) image. 
Thus, normalized "before" and "after" images can be subtractively 
compared, voxel - by - voxe 1 , to obtain a three-dimensional temporal 
difference image which represents the temporal differences 
between the before and after images. 
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Support for the assertion that temporal difference learning 
as described above is the technique to which the applicants are 
referring in the amended claims can be found in the specification 
at, for example, on page 7, line 21 to page 8, line 25, and on 
page 11, line 23 to page 13, line 10. 

This is much different from the method disclosed in 
Yanagita. As shown in his FIG. 2 and described at column 8, lines 
13-36, Yanagita performs basic subtraction processing between 
sequentially-acquired images. Subtraction processing provides a 
simple difference image, which can only be used to determine 
whether or not a change has occurred. This is not the same as the 
temporal difference image processing required by the amended 
claim. 

Yanagita is directed to a DISPLAY SYSTEM ("Image Displaying 
Apparatus")/ that can switch between temporally sequenced images 
that have been stored in memory, as well as between subtraction 
images. Subtraction images do show differences between sequential 
images, but the images must be interpreted by the observer as a 
vector or by additional processing. In other words, Yanagita' s 
subtraction images provide good graphics, but no metrics or other 
direct scientific information. 

In contrast, the applicants' method provides temporal 
information processing, which is capable of displaying temporal 
difference information over the associated image. This method 
provides tangible measures of changes that may have occurred over 
time. What Yanagita displays is open to interpretation and 
judgment, whereas the applicants' method provides temporal 
difference values that can be applied to limits, thresholds, 
ranges, rates of change, or other preset parameters that can be 
applied via analog or digital processing. 

An example which illustrates the difference between the 
methods might be helpful . Say a patient has a bleeder (aneurysm) 
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in an eye. A dye is injected into a nearby artery. Using 
Yanagita's method here results in a subtraction image which shows 
blood in the intraocular region. In contrast, the applicants' 
method provides temporal difference information which could 
indicate, for example, the rate and direction of blood flow, 
and/or the size (associated with rate) of the hole in the artery, 
and the 3D location around the vessel . 

In sum, there is a vast difference between Yanagita's 
approach and that of the applicants, with the applicants 
utilizing a much more complex and sophisticated analysis method 
(temporal difference information processing) , which yields 
substantially more information than does the Yanagita apparatus. 

A claim is anticipated when every one of its elements is 
disclosed in a single prior art reference. As has been shown 
above, Yanagita does not disclose the temporal difference 
processing required in the amended claim 1. Thus, Yanagita cannot 
and does not anticipate the amended claim 1, which is therefore 
allowable over Yanagita. 

Note that the specification has been amended to more 
explicitly describe aspects of the invention's use of temporal 
differences. The added text is either implicit in the discussion 
found in the original application, or concerns inherent 
characteristics of temporal difference learning which would be 
well known to those of ordinary skill in the art. No new matter 
has been added. 

Claims 9 and 10 

Each of claims 9 and 10 depend from the amended claim 1, and 
are therefore allowable on that basis. In addition, each of 
claims 9 and 10 has been amended to better clarify its 
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differences with respect to the cited art, such that these claims 
are also allowable on independent grounds. 

The amended claim 9 is directed to a method of creating the 
composite image of claim 1, which requires the determination of a 
" temporal difference image value based upon the temporal 
difference between said image intensity at said location in said 
first image and the respective intensity at said corresponding 
location in said second image." As discussed above, Yanagita 
fails to disclose the use of temporal difference image values are 
required by the amended claim 9. Claim 9 is thus allowable on 
this independent basis. 

The amended claim 10 also requires the use of temporal image 
differences, which are not disclosed or suggested in the Yanagita 
patent. Claim 10 is thus allowable on this independent basis. 

Rejections under §103 
Claims 2-3 and 5-6 

Claims 2-3 and 5-6 were rejected as obvious over Yanagita in 
view of Kawachi et al . 

Each of claims 2-3 and 5-6 depends from the amended claim 1, 
and are thus allowable along with claim 1. 

Claim 7 

Claim 7 was rejected as obvious over Yanagita and Kawachi 
and further in view of Wang. 

Claim 7 depends from the amended claim 1, and is thus 
allowable along with claim 1. 

Claim 8 

Claim 8 was rejected as obvious over Yanagita and Kawachi 
and further in view of Mitchell et al . 

Claim 8 depends from the amended claim 1, and is thus 
allowable along with claim 1. 
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Claims 4 and 19-21 

Claims 4 and 19-21 were rejected as obvious over Yanagita 
and Kawachi and further in view of Trezza. 

Claim 4 depends from the amended claim 1, and is thus 
allowable along with claim 1. 

Claim 19 is an independent claim, which has been amended to 
better clarify its differences with respect to the cited art. 

As amended, claim 19 is directed to a system for enhancing 
imagery of bodily tissues by relating earlier and later images, 
comprising : 

- an image processor, programmed to: 

- (a) receive the earlier and later images, 

- (b) register the earlier and later images by controlling 
an optical correlator to find a position of correlation between 
the earlier and later images, 

- (c) derive a composite image from the earlier and later 
images , 

- (d) compute temporal differences between the earlier and 
later images, and 

- (e) emphasize the temporal differences in said composite 
image ; and 

- an optical correlator coupled to said image processor and 
arranged to correlate said earlier and later images, and to 
output to said image processor a cross correlation image which 
is indicative of the position of correlation of the processed 
images . 

As noted above, the patent to Yanagita fails to disclose 
anything related to temporal difference image processing, 
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including the computing or the emphasizing of temporal 
differences between images as required by the amended claim 19. 
The citations to Kawachi and Trezza do nothing to cure these 
deficiencies, as they also fail to disclose a system capable of 
performing the recited functions. 

As such, the invention recited in the amended claim 19 would 
not have been obvious at the time the invention was made in view 
of Yanagita, Kawachi and Trezza. Claim 19 is thus allowable. 

As the limitations of claim 20 were largely incorporated 
into the amended claim 19, claim 20 has been canceled. 

The amended claim 19 is the parent of claim 21, which is 
therefore allowable along with claim 19. 

Claim 22 

Claim 22 was rejected as obvious over Yanagita, Kawachi, 
Tre z za and Wang . 

Claim 22 depends, from the amended claim 19, and is thus 
allowable along with claim 19. 

Claims 11-13 and 16 

Claims 11-13 and 16 were rejected as obvious over Yanagita 
in view of a patent to Gur et al . 

Claim 11 is an independent claim, which has been amended to 
better clarify its differences with respect to the cited art. 

As amended, claim 11 is directed to a method of creating a 
displayable composite mammographic image from a plurality of raw 
mammographic imagres, corresponding to earlier and later 
mammographic images, comprising the steps of: 

- (a) obtaining the earlier image of a region of tissue; 
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- (b) obtaining the later image of substantially the same region 
of tissue; 

(c) deriving a temporal difference image which represents 
changes between the earlier and later images; and 

- (d) combining at least one of the earlier and later images with 
the temporal difference image, to produce a composite image. 

As noted above, the patent to Yanagita fails to disclose or 
suggest anything related to temporal difference images, including 
the deriving or combining of such images as required by the 
amended claim 11. The citation to Gur does nothing to cure these 
deficiencies, as it also fails to disclose the use of the recited 
methods . 

As such, the invention recited in the amended claim 11 would 
not have been obvious at the time the invention was made in view 
of Yanagita and Gur. Claim 11 is thus allowable. 

Each of claims 12-13 and 16 depends from amended claim 11, 
and are thus allowable along with claim 11. 

Claim 14 

Claim . 14 ...was rejected as obvious over Yanagita, Gur and 

Wang . 

Claim 14 depends from the amended claim 11, and is thus 
allowable along with claim 11. 

Claim 15 

Claim 15 was rejected as obvious over Yanagita, Gur and 
Mitchell. 

Claim 15 depends from the amended claim 11, and is thus 
allowable along with claim 11. 
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Claim 17-18 

Claims 17-18 were rejected as obvious over Yanagita, Gur and 
Kawachi . 

Claims 17-18 depend from the amended claim 11, and are thus 
allowable along with claim 11. 

New claim 

A new claim, claim 23, has been added. Claim 23 depends from 
claim 4, and recites a combination of elements not disclosed in 
any of the cited art. 

All of the claims presently in the application are believed 
to be patentably distinct with respect to the cited art and to 
otherwise be in proper form for allowance. A Notice of Allowance 
is respectfully requested. 



Respectfully submitted, 




July 3, 2003 



Steven C. Patrick 
Registration No. 40,341 
Attorney for Applicant 



KOPPEL, JACOBS, PATRICK & HEYBL 
555 St. Charles Drive, Suite 107 
Thousand Oaks, California 91360 
(805) 373-0060 
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Attachment A: 

Amended specification with insertions and deletions indicated: 

Paragraph at page 2, lines 13-20: 

An image processing system and method visually documents 
historical changes in a medical image by processing and comparing 
historical with later images. A composite image is created which 
visually emphasizes the temporal differences between historical 
and later images. Preferably three-dimensional, digitized images, 
displayable in various projections, are stored for archival 
purposes on computer readable media. 

Paragraph at page 2, lines 21-32: 

To properly relate the historical and later images, an image 
processor preferably exploits an optical correlator to register 
the images accurately, by finding a transformation which produces 
a pre-determined degree of correlation between the images, then 
adjusting one image accordingly. The images are then compared, 
preferably volume element by volume element ("voxel by voxel") , 
to detect temporal differences between historical and later 
images. A composite image is displayed with synthetic colors or 
other visual clues to emphasize [apparent changes] the scalars in 
the imaged tissue, based on temporal differences between the 
historical and later images. 

Paragraph at page 5, line 16 through page 6, line 2: 

Next, in step 52, pathological feature data is preferably 
loaded by the image processor 24 (preferably from a data network 
but alternatively by retrieval from storage or through another 
input - channel ) . The pathological feature data is useful for 
narrowing the region of interest for image processing, and could 
be provided in any of several useful forms. Typically, a 
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physical region of interest (ROD in the historical image has 
been previously identified by a radiologist or other 
professional, and the coordinates of that region recorded in 



coordinates of the ROI are the pathological feature data. 
Alternatively, certain image shapes or characteristics, stored in 
a pathological image library, could be compared with the current 
image by image correlation to define the pathological region as 
described . below in connection with FIGs. 6a -6c. A further 
alternative is to employ known automated methods for the 
detection of abnormalities in digital mammograms. For example, 
see U.S. Patent No. [5, 4; 91, 627] 5, 491, 627 to Zhang et al . 
(1996) . The locations of any detected suspect lesions, as 
determined by such an automated method, could be used as the 
pathological feature data to identify and locate the ROI in the 



Paragraph between page 6, line 3 0 and page 7, line 9: 

Next, to obtain improved registration between the historical 
and new image, the image processor correlates the pathological 
feature set with the new image (step 58) , preferably by the 
method described below in connection with FIGs. 5a and 5b. The 
information gained from the correlation operations allows better 
registration of the new image with the historical imagery and 
defines an ROI in. the new image . This approach to 'better' 
registration provides for improved temporal difference 
resolution, and makes the associate measures more accurate within 
the region of interest . The ROI thus defined is then re-scanned 
(step 60) , preferably as described above, with scanning 
ultrasound equipment but with a finer level of resolution. Finer 
resolution can be obtained by using slower scan speeds, higher 
frequency ultrasound, finer and more numerous receive sensors, or 
other techniques as known in the art. The fine scan, being 



association with the historical image. 



In that case, the 



image . 
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preferably thus restricted to the ROI , can be executed in a 
shorter time than would be required to image the entire breast at 
a fine level of resolution. 

Paragraph between page 7, line 21 and page 8, line 4: 

After the historical and new images are aligned, they are 
compared in step 64 on a point-by-point basis, to obtain a 
temporal difference image which emphasizes the temporal changes 
which have occurred. A temporal difference image provides a 
scalar or reinforcement metric of learning, whereas a simple 
difference image simply determines whether or not a change has 
occurred. A temporal difference image is a reinforcement feedback 
signal indicating a scalar of how good/bad, how much change 

(vector) , or how strongly (acceleration) an action is happening. 
The scalars may represent, for example, a natural change, or the 
effectiveness of a treatment. For example, if digitized two 
dimensional images are considered, the images are preferably 
first normalized to a common average value, then corresponding 
pixels are subtracted and the temporal difference values stored 
in a corresponding position in a temporal difference image. 
Analogously, if three-dimensional imagery is available (which is 
greatly preferred) , voxels at corresponding locations in the 
early and later images are subtracted one from the other, and the 
result stored in a voxel of a temporal difference (volume) image. 

(A "voxel" is a unit of graphic information that defines a small 
volume element in three-dimensional space. It is the three- 
dimensional analog of a "pixel" which defines an area element in 
two- dimensional space.) Thus, normalized before and after images 
can be subtract ively compared, voxel -by -voxel , to obtain a three- 
dimensional temporal difference image which represents the 
temporal differences between the before and after images. 
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Paragraph at page 8, lines 5-25: 

As a result of step 64, at least three images are available 
to the image processor 24: an early (historical) image; a current 
image; and a temporal difference image. These images are then 
combined and/or selected for display (step 66) in various ways, 
as directed by user input from input device 32 . Finally, the 
resulting image or images are displayed (step 68) and preferably 
stored (step 70) and/or exported (step 72, preferably via the 
data network ) . It is highly [preferably] preferable that the 
imagery be coded to emphasize [visual] temporal differences 
between the earlier and later images. For example, the temporal 
difference image can be coded in a distinct color for display 
based on correlation output values , then superimposed on the 
early image, the later image, or a composite. Various other 
conventional methods could be equivalently employed to visually 
emphasize temporal image differences. For example, in one 
variation the temporal difference image can be displayed in 
intermittent, flickering display superimposed on either of the 
historical or current images. Significantly, the invention easily 
detects temporal differences between the earlier and later 
images, visually displays the images, and visually emphasizes 
these temporal differences. 

Paragraph at page 9, lines 16-32: 

The success of the above described procedure depends in part 
upon the accuracy with which the earlier and later images are 
registered (in steps 56 and 62 of FIG. 2a and 2b ) before 
comparison. Slight movement or deformation of the breast tissue 
or the scanning apparatus is to be expected between the earlier 
and later scans. The movement may include translation, rotation 
about any axis, or slight compression or expansion (in addition 
to biological tissue changes) . To adequately register the images 
in step 56 or 62 , therefore, a computationally practical and fast 
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method of registration is preferred. 



A preferred method of 



registration takes advantage of the specific computational 
abilities of an optical correlator (discussed in detail below in 
connection with FIG. 9) . This preferred method (suitable for use 
in step 64 of FIG. [2a] 2b) is best described with reference to 
an example of a particular coordinate system, to aid in 
visualization . 

Paragraph at page 12, lines 25-37: 

As an alternative to collapsing the images by simple 
projection, as described above, a three-dimensional image can be 
processed to create a "discriminate filter." Such filters are 
known and facilitate registration by providing a correlation 
filter which a composite of many images which differ by rotation 
about an axis. This allows testing of multiple, rotated slices 
simultaneously, by correlation with a composite discriminate 
filter. Once a discriminate filter is determined with acceptable 
correlation (at least [equa] equal to a pre-defined level of 
correlation) an individual filter is then selected from the set 
of slices which was used to compose the composite. In some 
cases, this method can greatly accelerate the search for best 
correlation . 

Paragraph at page 18, lines 3-31: 

After the earlier and later ROI images are adjusted to best 
register them, they can be either combined by adding, voxel -by- 
voxel, or compared by subtracting, voxel -by -voxel . Temporal 
differences [Differences] between the images are easily 
highlighted by changing the color at voxels which show high 
variation between the dual images. FIG. 8 shows an example of 
one typical display of a combined image (in this case, a 
composite of earlier and later images, with temporal differences 
highlighted) . The outline of the breast 22 is shown, with 
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highlighted suspected lesion 200 (simplified for illustration) as 
revealed by a recent ultrasonogram. Other more static regions 
of density 2 02 and 2 03 are shown, which would suitably be 
displayed in a neutral color or grey scale. The earlier lesion 
region 206 is surrounded by the later, larger lesion 200. 
Suitably the later region 200 would be highlighted, by example by 
display in a conspicuous color such as pink. The smaller lesion 
206 (visible in the earlier historical image) would preferably be 
coded in a different color (for example, green) . This three 
dimensional image, easily digitized and archived on computer 
readable media, provides easily readable documentation of the 
historical image changes, and can be retrieved for evidence of 
tissue changes. The image processor can also extract slices such 
as that cut by imaginary plane 302, and display the slice as a 
two-dimensional section for detailed inspection. Preferably, the 
processor is programmed to respond to user input so that any 
slice can be selected for display, or the user can sweep through 
multiple slices to view details of interest. 

Paragraph between page 21, line 24 and page 22, line 1: 

When the input the filter images have been written to the 
input ..and filter ports .210 and .-212 , . „ the optical correlator 
produces an output image which is a two dimensional output 
correlation pattern having an optical peak or peaks (bright spot) 
at the position of greatest correlation between the collapsed 
sonographic image and the radiographic image. The degree of 
correlation is indicated by the intensity of the output signal. 
The position of the output peak on the two-dimensional matrix of 
the correlator output CCD indicates the translations or shifts of 
the images relative to one another. The output image is read from 
the output photodetector (CCD) 214 by the image processor 24 in 
the conventional manner, typically by shifting the CCD voltage 
values out sequentially in rows (or columns) and then digitizing, 
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the output levels. The peak amplitudes of the 2D array provide 
the extent of correlation (degree of correlation) between the 
images. This scalar information is a measure of the temporal 
difference, and can be used as a measure of knowledge between 
images and supports actions that lead to a successful treatment 
or evaluation . 

Paragraph at page 29, lines 1-14: 

An image processing system and method visually documents and 
displays changes between historical and later mammographic 
images, preferably in three dimensions. A composite image is 
created which visually emphasizes temporal differences between 
the historical and later images. Preferably three-dimensional, 
digitized images, displayable in various projections, are stored 
for archival purposes on computer readable media. An image 
processor preferably exploits an optical correlator to register 
the historical and later images accurately and provide 
correlation values as temporal scalars of the differences . The 
registered images are then compared, voxel -by -voxel , to detect 
temporal differences. The composite image is displayed with 
synthetic colors or other visual clues to emphasize apparent 
changes, (for example, tumor growth or shrinkage) . 
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Amended claims with insertions and deletions indicated: 

1. (amended) A method of visually documenting historical 
changes in biological tissue, comprising the steps of: 

(a) obtaining a first image of a region of tissue; 

(b) after obtaining said first image, obtaining a second 
image of said region of tissue; 

(c) digitally storing said first and second images as 
digitized first and second images; 

(d) spatially adjusting at least one of said first and 
second digitized images to spatially register said images so that 
corresponding features in both images are mapped to corresponding 
positions; and 

(e) creating from said first and second digitized images a 
derived image which visually emphasizes temporal differences 
between said first and second images, thereby visually 
emphasizing historical changes between said images. 

9. (amended) The method of claim 1, wherein said step of 
creating a composite image comprises: 

comparing an image intensity at a location in said 
first image with a respective intensity at a corresponding 
location in said second image, and 

determining a temporal difference' image value based 
upon the temporal difference between said image intensity at said 
location in said first image and the respective intensity at said 
corresponding location in said second image. 

10. (amended) The method of claim 1, wherein said composite 
image visually emphasizes temporal image differences by 
representing various regions of said composite image in synthetic 
colors, based upon temporal image differences between the first 
and second images. 
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11. (amended) A method of creating a displayable composite 
mammographic image from a plurality of raw mammographic images, 
corresponding to earlier and later mammographic images, 
comprising the steps of : 

(a) obtaining the earlier image of a region of tissue; 

(b) obtaining the later image of substantially the same 
region of tissue; 

(c) deriving a temporal difference image which 
represents changes between said earlier and later images; and 

(d) combining at least one of said earlier and later 
images with said temporal difference image, to produce a 
composite image. 

16. (amended) The method of claim 11, wherein said step of 
creating a composite image comprises: 

spatially adjusting at least [on] one of said earlier 
and later images to aid in registering said images. 

19. (amended) A system for enhancing imagery of bodily 
tissues by relating earlier and later images, comprising: 

an image processor, programmed to: (a) receive said 
earlier and later images, (b) register the earlier and later 
images by controlling an optical correlator to find a position of 
correlation between said earlier and later images, [and] (c) 
derive a composite image from the earlier and later images , (d) 
compute temporal differences between said earlier and later 
images, and (e) emphasize said temporal differences in said 
composite image ; and 

an optical correlator coupled to said image 
processor and arranged to correlate said earlier and later 
images, and to output to said image processor a cross 
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correlation image which is indicative of the position of 
correlation of the processed images. 

23. The method of claim 4, wherein said step of creating a 
composite image comprises: 

comparing an image intensity at a location in said 
first image with a respective intensity at a corresponding 
location in said second image, and 

determining a temporal difference image value based 
upon said correlation output values. 



